Information processing apparatus and information processing method

ABSTRACT

There is provided an information processing apparatus including an image acquisition unit configured to acquire a dish image obtained by shooting a single or multiple dishes, and a first dish recognition unit configured to recognize the single or multiple dishes included in the dish image with reference to dish data selected, from dish data registered in advance, based on a condition regarding at least one of a person relating to the dish image, a shooting environment of the dish image, a shooting place of the dish image, and a shooting time of the dish image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2012-238768 filed Oct. 30, 2012, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an information processing apparatusand an information processing method.

Such a technique is gaining widespread use that electrically records,for daily dietary management and the like, information regarding mealsthat a user takes and, for example, calculates and provides nutritionand calories to the user as information. In such a technique, a usergenerally inputs information regarding a meal by use of a terminalapparatus that the user uses, and transmits the information to a server.However, as a method for simplifying input of information, such atechnique is proposed that uses an image shot by a user as inputs ofinformation regarding a meal.

For example, JP 2011-28382A describes a technique for detecting an areafrom an image of a meal shot by a user, for each dish (such as rice,miso soup, stir-fried vegetables, and coffee), and recognizing throughtemplate matching and the like what each dish shown in the area is. As atemplate used for recognition, the following two images can be used: animage of a dish that a user has previously taken, and an image of a mealthat is registered as a standard.

If an image of a meal that a user has previously taken is used as atemplate for recognition in this way, it becomes possible to accuratelyrecognize dishes that the user takes on a daily basis. Use of an imageof a meal that is registered as a standard with the above-describedimage allows the user to recognize even dishes that the user does nottake on a daily basis.

SUMMARY

The technique described in JP 2011-28382A allows more dishes to beaccurately recognized with increase in images of dishes registered astemplates. However, the more images (that are candidates for dishesshown in an image) are targets for matching, the heavier loads areplaced in performing a process such as template matching used for dishrecognition. That is, it is difficult that a technique as described inJP 2011-28382A recognizes a dish more easily and decreases processingloads at the same time.

The present disclosure therefore proposes an information processingapparatus and an information processing method that are novel andimproved, and can maintain the accuracy of recognizing a dish from animage and decrease processing loads by limiting data that is referred toin recognition.

According to an embodiment of the present disclosure, there is providedan information processing apparatus including an image acquisition unitconfigured to acquire a dish image obtained by shooting a single ormultiple dishes, and a first dish recognition unit configured torecognize the single or multiple dishes included in the dish image withreference to dish data selected, from dish data registered in advance,based on a condition regarding at least one of a person relating to thedish image, a shooting environment of the dish image, a shooting placeof the dish image, and a shooting time of the dish image.

According to another embodiment of the present disclosure, there isprovided an information processing apparatus including acquiring a dishimage obtained by shooting a single or multiple dishes, and recognizingthe single or multiple dishes included in the dish image with referenceto dish data selected, from dish data registered in advance, based on acondition regarding at least one of a person relating to the dish image,a shooting environment of the dish image, a shooting place of the dishimage, and a shooting time of the dish image.

It is selected on the basis of a person relating to a dish image, ashooting environment of the dish image, a shooting place of the dishimage, a shooting time of the dish image, or the like which dish datashould be referred to. It is hereby possible to selectively refer todish data relating to the dish image to perform dish recognition. It ispossible to maintain the accuracy of dish recognition and decreaseprocessing loads by selectively referring to dish data that is estimatedto be useful for dish recognition.

According to one or more of embodiments of the present disclosure, it ispossible to maintain the accuracy of recognizing a dish from an image,and decrease processing loads by limiting data that is referred to inrecognition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afirst embodiment of the present disclosure;

FIG. 2 is a diagram for schematically describing a dish recognitionprocess in the first embodiment of the present disclosure;

FIG. 3A is a diagram illustrating a first example of metadata of a dishimage in the first embodiment of the present disclosure;

FIG. 3B is a diagram illustrating a first example of informationassociated with dish data in the first embodiment of the presentdisclosure;

FIG. 4 is a flowchart illustrating an example of a dish recognitionprocess in the first embodiment of the present disclosure;

FIG. 5A is a diagram illustrating a second example of metadata of a dishimage in the first embodiment of the present disclosure;

FIG. 5B is a diagram illustrating a second example of informationassociated with dish data in the first embodiment of the presentdisclosure;

FIG. 6 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to asecond embodiment of the present disclosure;

FIG. 7 is a diagram illustrating an example of information associatedwith dish data in the second embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating an example of a dish recognitionprocess in the second embodiment of the present disclosure;

FIG. 9 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to athird embodiment of the present disclosure;

FIG. 10 is a diagram illustrating an example of information associatedwith dish data in the third embodiment of the present disclosure;

FIG. 11 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afourth embodiment of the present disclosure;

FIG. 12 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afifth embodiment of the present disclosure;

FIG. 13 is a diagram for describing an example of additionalregistration of dish data in the fifth embodiment of the presentdisclosure;

FIG. 14 is a block diagram illustrating a schematic functionalconfiguration of a system according to a sixth embodiment of the presentdisclosure; and

FIG. 15 is a block diagram for describing a hardware configuration of aninformation processing apparatus.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The description will be made in the following order.

1. First Embodiment 1-1. Functional Configuration 1-2. Overview of DishRecognition Process 1-3. First Example of Dish Recognition Process 1-4.Second Example of Dish Recognition Process 2. Second Embodiment 3. ThirdEmbodiment 4. Fourth Embodiment 5. Fifth Embodiment 6. Sixth Embodiment7. Hardware Configuration 8. Supplement 1. First Embodiment 1-1.Functional Configuration

FIG. 1 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afirst embodiment of the present disclosure. With reference to FIG. 1, aninformation processing apparatus 100 includes an image acquisition unit110, an information extraction unit 120, a dish recognition unit 130, adatabase 140, and a result output unit 150. The information processingapparatus 100 performs a process of acquiring a dish image as an input,and recognizing a dish included in the dish image.

The information processing apparatus 100 may be, for example, a terminalapparatus that is used by a user, or a server that communicates with aterminal apparatus which is a client and provides a service to a user.The terminal apparatus may be, for example, a personal computer (PC)such as a tablet PC, a notebook PC, and a desktop PC, a mobile phone(smartphone), a media player, or a game device. A function of the servermay be realized, for example, by a single server apparatus, or multipleserver apparatuses that are connected through a wired or wirelessnetwork. The terminal apparatus and the server apparatus may berealized, for example, by using a hardware configuration of theinformation processing apparatus, which will be described below.

The image acquisition unit 110 acquires a dish image obtained byshooting a single or multiple dishes. For example, if the informationprocessing apparatus 100 is a terminal apparatus, the image acquisitionunit 110 is realized as a camera (imaging unit) that shoots a dishimage. Alternatively, the image acquisition unit 110 may be realized asa communication device that receives a dish image shot by an apparatushaving a camera installed therein. In that case, the communicationapparatus receives a dish image from another apparatus that isconnected, for example, through a home network and owned by a user.Meanwhile, if the information processing apparatus 100 is a server, theimage acquisition unit 110 is realized, for example, as a communicationdevice that receives a dish image from a client terminal apparatusconnected through a network.

The dish image means an image obtained by shooting, for example, a mealthat a user takes. For example, a light meal includes a single dish in adish image. A usual meal includes multiple dishes such as a staple food,a main dish, and a side dish in a dish image. The information processingapparatus 100 recognizes a dish included in a dish image, which isnamely a dish included in a meal which a user takes, through a processof the dish recognition unit 130, which will be described below.

A dish image entails metadata in the present embodiment. The metadataincludes information indicating, for example, a person relating to thedish image, a shooting environment of the dish image, a shooting placeof the dish image, or a shooting time of the dish image. Such kinds ofinformation may be automatically set, for example, by a terminalapparatus that shoots the dish image. Alternatively, the information maybe set in accordance with an input operation of a user. For example, themetadata may also be recorded in accordance with a compatible formatsuch as the Exchangeable image file format (Exif). Alternatively, themetadata may be recorded in a format unique to an application thatprovides analysis of the dish image.

The information extraction unit 120 extracts information regarding atleast one of the person relating to the dish image, the shootingenvironment of the dish image, the shooting place of the dish image, andthe shooting time of the dish image from the metadata of the dish imageacquired by the image acquisition unit 110, and provides the extractedinformation to the dish recognition unit 130. The information extractionunit 120 is realized by a processor such as a central processing unit(CPU) operating in accordance with a program stored in a memory.

The dish recognition unit 130 recognizes a single or multiple dishincluded in the dish image with reference to dish data selected, fromdish data registered in advance, on the basis of a condition regardingat least one of the person relating to the dish image, the shootingenvironment of the dish image, the shooting place of the dish image, andthe shooting time of the dish image. Information used for setting thecondition is extracted from the metadata of the dish image by theinformation extraction unit 120 in the present embodiment. The dish datais stored in the database 140. The dish data is data corresponding toeach dish that may be included in the dish image. Processing loads indish recognition are decreased by selecting dish data that is referredto by the dish recognition unit 130 in a process of dish recognition onthe basis of a predetermined condition. Additionally, a detailed processof recognizing a dish with reference to dish data will be describedbelow. The dish recognition unit 130 is also realized by a processorsuch as a CPU operating in accordance with a program stored in a memory.

The “person relating to a dish image” herein includes, for example, auser who shoots the dish image, and a person who is with the user whenthe dish image is shot, which namely means a person who takes a mealshown in the dish image with the user. The “shooting environment of adish image” includes, for example, luminous intensity in theneighborhood at the time of shooting the dish image, colors of thebackground table and a tray, and conditions of illumination (that meanwhether the illumination is natural light, light of a fluorescent lamp,light of an electric bulb, or the like). The “shooting place of a dishimage” includes, for example, coordinates (latitude and longitude) of aplace in which the dish image is shot, and a specific name of the place(such as an office, a home, and a restaurant). Additionally, theshooting place of a dish image does not necessarily have to beidentified in the form of a point, but may be identified, for example,in the form of an area having a predetermined size, an administrativedistrict, and a name of the place. The “shooting time of a dish image”means a time at which the dish image is shot. The time may include adate. Dish data that is referred to by the dish recognition unit 130 isselected in accordance with a condition based on subsidiary informationof the dish image as described above in the present embodiment.

The database 140 stores dish data that is referred to in a process ofdish recognition performed by the dish recognition unit 130. The dishdata is stored in association with information such as the person, theshooting situation, the shooting place, and the shooting time. The dishrecognition unit 130 selects dish data that is referred to in a processof dish recognition, on the basis of the above-described informationassociated with the dish data, and a condition that is set by theinformation extraction unit 120 on the basis of information extractedfrom metadata of the dish image. The database 140 is realized, forexample, by a storage device of the information processing apparatus100. Alternatively, the database 140 is realized by an external storagedevice of the information processing apparatus 100. The dish recognitionunit 130 may access the database 140 via a network.

The result output unit 150 outputs information regarding a result ofdish recognition performed by the dish recognition unit 130. Forexample, if the information processing apparatus 100 is a terminalapparatus, the result output unit 150 may be realized as an outputdevice such as a display and a speaker that outputs the informationregarding the recognition result in the form of an image or sounds.Alternatively, the result output unit 150 may be realized as acommunication device that transmits the information regarding the dishrecognition result to another apparatus. If the information processingapparatus 100 is a server, the result output unit 150 outputs theinformation regarding the dish recognition result to, for example, aclient terminal apparatus. The result output unit 150 may output theinformation regarding the dish recognition result to, for example, anapparatus that outputs information to a user, or an apparatus that usesthe dish recognition result for further analysis and data accumulation.

Information such as nutrition and calories regarding each dish has beenalready provided on the basis of a material for the dish and a methodfor cooking the dish. Such kinds of information can be stored in thedatabase 140, and also be acquired from an appropriate service on anetwork. Accordingly, once a dish included in a dish image isrecognized, it is possible to calculate, on the basis of information ofthe recognized dish such as nutrition and calories, the nutrition andthe calories of the meal that a user takes, and provide the informationto the user via the result output unit 150. Such kinds of informationregarding the recognition result may be provided to a user each time theuser takes a meal. Alternatively, the information regarding therecognition result may be accumulated as a log, further analyzed asnecessary, and provided to the user.

1-2. Overview of Dish Recognition Process

FIG. 2 is a diagram for schematically describing a dish recognitionprocess according to the first embodiment of the present disclosure.With reference to FIG. 2, the dish recognition unit 130 of theinformation processing apparatus 100 recognizes dishes 1012 included ina dish image 1010 by matching the dish image 1010 with dish data 1020.In the illustrated example, rice 1012 a, a salad 1012 b, an egg 1012 c,a fish 1012 d, and miso soup 1012 e are recognized as the dishes 1012.

The dish image 1010 is an image obtained by shooting, for example, ameal that a user takes, and acquired by the image acquisition unit 110.Meanwhile, the dish data 1020 is data corresponding to each dish thatmay be included in the dish image 1010, registered in advance, andstored in the database 140. The dish recognition unit 130 matches thedish image 1010 with the dish data 1020. For example, if the dish data1020 is data of an image showing each dish, the dish recognition unit130 performs matching such as template matching on the images. Variousknown techniques as described in, for example, JP 2011-28382A can beused for matching.

As an example, the dish image 1010 herein matches with the dish data1020 in a process performed by the dish recognition unit 130. However,the dish recognition unit 130 may recognize the dishes 1012 included inthe dish image 1010 through a process other than matching. For example,the dish recognition unit 130 may use the selected dish data 1020 as alearning sample to recognize the dishes 1012 through object recognition.

The above-described processes of matching and object recognition bothplace heavy loads. Thus, heavier loads are placed in a process performedby the dish recognition unit 130 with increase in the volume of dishdata 1020 registered with the database 140, for example. To thecontrary, the accuracy of dish recognition performed by the dishrecognition unit 130 is reduced more with decrease in the registeredvolume of dish data 1020 because fewer dishes can be recognized.

In the present embodiment, the dish recognition unit 130 thereforeselects dish data 1020 that is referred to in a process of dishrecognition on the basis of a condition regarding a person relating tothe dish image 1010, a shooting situation of the dish image 1010, ashooting place of the dish image 1010, a shooting time of the dish image1010, or the like. It is hereby possible to maintain the accuracy ofdish recognition performed by the dish recognition unit 130, anddecrease processing loads by limiting the dish data 1020 that isreferred to.

The process of dish recognition in the present embodiment will befurther described below with reference to specific examples of acondition indicated by metadata of the dish image 1010, and informationassociated with the dish data 1020.

1-3. First Example of Dish Recognition Process

FIGS. 3A and 3B are diagrams illustrating a first example of metadata ofa dish image and information that is associated with dish data in thefirst embodiment of the present disclosure, respectively. FIG. 3Aillustrates pieces of metadata 1030 a to 1030 c as examples of metadata1030 of a dish image. FIG. 3B illustrates pieces of dish data 1020 a,1020 b, 1020 p, and 1020 q as examples of dish data 1020.

The metadata 1030 illustrated in FIG. 3A corresponds to dish images(001.jpg to 003.jpg). Additionally, the dish image is not limited to animage in the jpg format. In the first example, a date (Date), a time(Time), a place (Place), a person who shoots a dish (By), and a personwith the person who shoots the dish (With) are set as items of themetadata 1030. For example, the metadata 1030 a of the dish image001.jpg indicates that the dish image 001.jpg is shot by John at 12:10on Friday, Oct. 12, 2012 in an office with Paul and George.

Such kinds of metadata 1030 may be set on the basis of informationautomatically detected by a terminal apparatus that shoots the dishimage. For example, position information of a shooting place that isacquired by using a shooting date, a shooting time, a global positioningsystem (GPS), and the like is generally known as information that isadded to an image as metadata by using a compatible format such as theExif. Furthermore, information regarding a user who shoots the dishimage and another person with the user when the dish image is shot canbe automatically added to the image as metadata on the basis of, forexample, user information registered with the apparatus, and positioninformation provided from a terminal apparatus held by the other person.

Alternatively, the metadata 1030 may be set in accordance with ashooting time of the dish image or a later input operation of the user.In that case, the user sets metadata through an input operation such asso-called tagging with respect to, for example, a shooting place of thedish image, the user who shoots the dish image, and a person who is withthe user when the dish image is shot. The metadata 1030 may include bothan item that is set through an input operation of the user as describedabove and an item that is automatically detected. Alternatively, themetadata 1030 may include any one of the items.

Additionally, information of each item that is included in the metadata1030 has been specifically shown for convenience of explanation, whichdoes not intend to limit a form of metadata. For example, timeinformation may be recorded along with a date, position information maybe recorded in the form of coordinates (latitude and longitude), and aperson who shoots an image and a person who is with the person shootingthe image may be recorded in the form of user ID. The metadata 1030 doesnot have to include all the items shown in the figure, and may includeat least one of the items.

The dish data 1020 illustrated in FIG. 3B is recorded for each dish. Inthe illustrated example, the dish data 1020 corresponds to pieces ofdish image data (P001.jpg, P002.jpg, P011.jpg, and P021.jpg).Additionally, the dish image data is not limited to image data in thejpg format. In the first example, the dish data 1020 is associated withinformation such as a person who owns the data (Whose data?), a timezone in which a dish is taken (When?), a place in which the dish istaken (Where?), and a person who takes the dish together (With whom?).For example, the dish data 1020 a indicating a dish, “rice,” isassociated with information indicating that John has the dish data 1020a, and “rice” is a dish taken in an office as a lunch on a weekday withcoworkers (such as Paul and George).

The dish data 1020 does not necessarily have a one-to-one correspondenceto the dishes. That is, multiple pieces of dish data 1020 may be setwith respect to the same dish. For example, the dish indicated by thedish data 1020 a means not just “rice,” but “rice taken as a lunch in anoffice on a weekday with coworkers.” In addition, there may be dish data1020 p regarding “rice” indicating “rice taken as a dinner on a weekendat home with a family.” This is because the dish images may havedifferent characteristics due to, for example, time zones for meals andanother situation, though both of the images include the same dishes.For example, a bowl for “rice” taken as a lunch on a weekday in acafeteria of an office has a different shape and a different color froma bowl for “rice” taken as a dinner on a weekend, and luminous intensityin the neighborhood is also different. Consequently, the dish imageshave different characteristics. Thus, it is preferable to preparedifferent kinds of dish data 1020 for the two types of “rice,” andrecognize that both of the dish images include “rice.”

Such kinds of dish data 1020 may be registered through an explicit inputoperation of a user. For example, a user manually inputs a dish areaincluded in a shot dish image, and a dish included in the area. The dishdata 1020 may be registered on the basis thereof. In that case,information associated with the dish data 1020 may be set on the basisof information regarding, for example, a shooting date and time and ashooting place of the original dish image. Alternatively, the dish data1020 may be distributed to a user who satisfies a predeterminedcondition. For example, the dish data 1020 regarding dishes on a menu ofa cafeteria of an office may be distributed to a worker in the office,or the dish data 1020 regarding dishes on a menu of a restaurant may bedistributed to a user who has been registered as a member for therestaurant.

Additionally, each item included in the dish data 1020 has also beenspecifically shown in the figure for convenience of explanation, whichdoes not intend to limit the form of the dish data. For example,position information may be recorded in a range specified by coordinates(latitude and longitude), or a person may be recorded in the form ofuser ID. The dish data 1020 does not have to include all the informationshown in the figure, and may include at least one piece of information.However, the metadata 1030 and the dish data 1020 include at least onecorresponding piece of information. For example, if the metadata 1030includes only information regarding a date (Date) and a time (Time), thedish data 1020 is associated with at least information regarding a timezone in which the dish is taken (When?). For example, if the dish data1020 is associated with only information regarding a place in which thedish is taken (Where?), the metadata 1030 includes at least informationregarding a place (Place).

Next, with reference to a flowchart illustrated in FIG. 4, an example ofa dish recognition process using the metadata 1030 and the dish data1020, as described above, will be described.

FIG. 4 is a flowchart illustrating an example of a dish recognitionprocess in the first embodiment of the present disclosure. In theillustrated example, the dish recognition unit 130 performs loopprocessing in units of the dish data 1020 when recognizing a dishincluded in a dish image (step S101).

The dish recognition unit 130 determines whether the dish data 1020satisfies a condition indicated by metadata of the dish image (stepS103). If the dish data 1020 satisfies the condition (YES), the dishrecognition unit 130 uses the dish data 1020 for matching with the dishimage (step S105). To the contrary, if the dish data 1020 does notsatisfy the condition (NO), the dish recognition unit 130 does not usethe dish data 1020 for matching, but proceeds to process the next dishdata 1020 (step S101).

The above-described process that is performed by using the metadata 1030and the dish data 1020 illustrated in FIGS. 3A and 3B will be furtherdescribed.

As an example, let us assume that the image acquisition unit 110acquires the dish image (001.jpg) that entails the metadata 1030 a. Inthat case, the metadata 1030 a indicates conditions that the dish image001.jpg is “shot by John” (person relating to the image) “around thenoon on Friday (weekday)” (shooting time) “in an office” (shootingplace) “with Paul and George.” Accordingly, the dish recognition unit130 selects, as dish data used for matching, the dish data 1020registered in association with information that satisfies theconditions. Additionally, all the conditions do not have to be used forextraction of the dish data 1020. A part of the conditions, which means,for example, one or two of a shooting time, a shooting place, and aperson relating to the image may be used.

The pieces of dish data 1020 a and 1020 b of the dish data 1020 are eachassociated with information indicating that a meal is taken “as a lunchon a weekday” “in an office” “with coworkers (including Paul andGeorge).” Thus, such kinds of dish data satisfy the conditions indicatedby the metadata 1030 a. To the contrary, the pieces of dish data 1020 pand 1020 q do not satisfy the conditions indicated by the metadata 1030a because the pieces of data 1020 p and 1020 q indicate that a meal istaken “as a lunch on a weekend” or “at home.”

Thus, determination in step S103 shows YES with respect to the pieces ofdish data 1020 a and 1020 b, and NO with respect to the pieces of dishdata 1020 p and 1020 q. Consequently, the pieces of dish data 1020 a and1020 b are used for matching in step S105, while the pieces of dish data1020 p and 1020 q are not used for matching.

As the associated information indicates, the pieces of dish data 1020 pand 1020 q are a piece of data indicating a meal taken as a dinner athome with a family and a piece of data indicating a meal taken as alunch on a weekend in a restaurant with a friend, respectively. Thus, ifthe pieces of data are used for matching with the dish image 001.jpg(that is an image of a meal taken as a lunch on a weekday in an officewith a coworker), there is little possibility that the pieces of datamatch with the dish image 001.jpg. It is possible to maintain theaccuracy of recognition and decrease the processing loads by performingdish recognition with the dish data 1020 excluded from the referencetargets.

In the illustrated examples, the pieces of dish data 1020 a, 1020 b,1020 p, and 1020 q are all pieces of data regarding John. Accordingly, acondition of a person who shoots the dish image is not useful forselecting the dish data 1020. In another example, if the database 140 ofthe information processing apparatus 100, which is, for example, aserver, stores data regarding multiple users, the dish recognition unit130 may selectively refer to the dish data 1020 that is indicated asbeing data regarding a person who shoots the dish image indicated by themetadata 1030.

1-4. Second Example of Dish Recognition Process

FIGS. 5A and 5B are diagrams illustrating metadata of a dish image inthe first embodiment of the present disclosure and a second example ofinformation associated with dish data, respectively. FIG. 5A illustratespieces of metadata 1032 a to 1032 c as examples of metadata 1032 of thedish images. FIG. 5B illustrates pieces of dish data 1022 a, 1022 b,1022 p, and 1022 q as examples of dish data 1022.

The metadata 1032 illustrated in FIG. 5A corresponds to the dish images(001.jpg to 003.jpg). The dish image is not limited to an image in thejpg format. In the second example, for example, a date (Date), a time(Time), a shutter speed (Shutter Speed), and use of a flashlight(Flash?) are set as items of the metadata 1032. For example, themetadata 1032 a of the dish image 001.jpg indicates that the dish imageis shot at 12:10 on Friday, Oct. 12, 2012 at a shutter speed of 1/240seconds without a flashlight.

Such kinds of metadata 1032 may be set, for example, on the basis ofinformation automatically detected by a terminal apparatus that shootsthe dish image. An item that is included in the metadata 1032 andindicates a shooting environment of the dish image is generally recordedby using a compatible format such as the Exif. Thus, as an example, theinformation extraction unit 120 may extract, from the Exif data added tothe dish image, information indicating a condition that is used for thedish recognition unit 130 to select dish data.

The dish data 1022 illustrated in FIG. 5B are set for each dish. In theillustrated example, the dish data 1022 corresponds to each piece ofdish image data (P001.jpg, P002.jpg, P011.jpg, and P021.jpg). The dishimage data is not limited to an image data in the jpg format. In thesecond example, the dish data 1022 is associated with information suchas a person who owns the data (Whose data?), a shutter speed (ShutterSpeed), and use of a flashlight (Flash?). For example, if the dish imagedata is an image that is cut away from a previous dish image, such akind of information is set on the basis of information such as the Exifthat indicates a shooting situation of the original image.

In the second example, the dish data 1022 does not necessarily have aone-to-one correspondence to dishes. That is, multiple pieces of dishdata 1022 may be set with respect to the same dish. For example, thepieces of dish data 1022 a and 1022 p are both pieces of dish dataregarding “rice.” However, since the dish data 1022 a is an image thatis shot at a high shutter speed without a flashlight, the dish data 1022a is estimated, for example, to be cut away from an image shot in alight cafeteria of an office in a daytime. Meanwhile, since the dishdata 1022 p is an image shot at a low shutter speed with a flashlight,the dish data 1022 p is estimated, for example, to be cut away from animage shot at night at a gloomy home. Thus, it is preferable to preparedifferent pieces of dish data 1022 for the two types of “rice,” andrecognize that the dish images including both the types include “rice.”

By use of the metadata 1032 and the dish data 1022 illustrated in FIGS.5A and 5B, execution of the process of dish recognition illustrated inFIG. 4 will be further described.

As an example, let us assume that the image acquisition unit 110acquires the dish image (002.jpg) that entails the metadata 1032 b. Inthat case, the metadata 1032 a indicates that the dish image 002.jpg isshot at a shutter speed of 1/120 seconds with a flashlight. Accordingly,the dish recognition unit 130 selects, as dish data used for matching,the dish data 1022, which is registered in association with informationcorresponding to the above-described conditions. Additionally, all ofthe conditions do not have to be used for extraction of the dish data1022. A part of the conditions, which means, for example, any of ashutter speed and use of a flashlight may be used.

The dish data 1022 p of the dish data 1022 is associated withinformation indicating that the dish data 1022 p is shot at a shutterspeed ( 1/125 seconds) approximating to the shutter speed ( 1/125seconds) of the metadata 1032 b with a flashlight. Thus, the dish data1022 p satisfies the conditions indicated by the metadata 1032 b. To thecontrary, the pieces of dish data 1022 a, 1022 b, and 1022 q have a muchhigher shutter speed ( 1/250 seconds) or a much lower shutter speed (1/60 seconds) than the dish image, or are shot without a flashlight,which do not satisfy the conditions indicated by the metadata 1032 a.

Thus, determination in step S103 shows YES with respect to the dish data1022 p, and NO with respect to the pieces of dish data 1022 a, 1022 b,and 1022 q. Thus, the dish data 1022 p is used for matching in stepS105. The pieces of dish data 1022 a, 1022 b, and 1022 q are not usedfor matching.

The pieces of dish data 1022 a and 1022 b are pieces of data of dishesthat are taken in a very light environment, as indicated by theassociated information. To the contrary, the dish data 1022 q is a pieceof data of a dish that is taken in a very dark environment. Thus, if thepieces of dish data are used for matching with the dish image 002.jpg(image shot in a gloomy environment), there is little possibility thatthe pieces of dish data match with the dish image 002.jpg. It ispossible to maintain the accuracy of recognition and decrease processingloads by performing dish recognition with the dish data 1022 excludedfrom the reference targets.

As shown by the above-described example, a certain latitude may beallowed in determining whether the dish data 1022 satisfies theconditions indicated by the metadata 1032. In the above-describedexample, the metadata 1032 b indicates that the dish image is shot at ashutter speed of 1/120 seconds, and the shutter speed associated withthe dish data 1022 p is 1/125 seconds. However, the dish data 1022 p istreated as satisfying the condition indicated by the metadata 1032 b.

2. Second Embodiment

FIG. 6 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to asecond embodiment of the present disclosure. With reference to FIG. 6,an information processing apparatus 200 includes an image acquisitionunit 110, a dish recognition unit 130, a database 140, and a resultoutput unit 150. Different from the information processing apparatus 100according to the first embodiment, the information processing apparatus200 does not include an information extraction unit 120. Thus, thepresent embodiment has a different process of extracting a condition forselecting dish data in the dish recognition unit 130, from the processin the first embodiment. The other configurations according to thepresent embodiment are substantially the same as the configurations ofthe first embodiment so that the repeated detailed description will beomitted.

In the present embodiment, the dish recognition unit 130 treats acharacteristic of a whole dish image as information regarding at leastone of a shooting environment of the dish image, a shooting place of thedish image, and a shooting time of the dish image. This is becauseimages that have the same shooting situation, shooting place, orshooting time can possibly have the similar characteristics of the wholedish images. For example, dish images obtained by shooting meals at thesame place (such as a cafeteria of an office, a home, and a restaurant)have, for example, the same tray, table, and tableware so that the dishimages have the similar characteristics as the whole dish images.

Even if places are not identified, a dish image obtained by shooting ameal in a relatively light environment (such as a cafeteria of an officeand an outdoor space) has a different characteristic as a whole dishimage from a characteristic of a dish image obtained by shooting a mealin a relatively dark environment (such as a home and an atmosphericrestaurant). Accordingly, it is possible to determine, on the basis ofthe characteristics of the whole dish images, in which environment thedish image is shot. For example, even meals in the same place may befurther reflected on characteristics of the whole dish images becauseluminous intensity is different owing to time zones.

In view of the above-described points, in the present embodiment, thedish recognition unit 130 compares a characteristic of a whole dishimage with a characteristic of a source image from which the image ofdish data is cut away, and selects dish data that is referred to, inaccordance with the result. More specifically, the dish recognition unit130 recognizes a dish included in a dish image by selectively referringto, as dish data, data of an image that is cut away from a source imagehaving a characteristic similar to a characteristic of the whole dishimage. The dish recognition process will be further described below withreference to specific examples of dish data.

FIG. 7 is a diagram illustrating an example of information associatedwith dish data in the second embodiment of the present disclosure. FIG.7 illustrates pieces of dish data 2020 a, 2020 b, 2020 p, and 2020 q asexamples of dish data 2020.

The dish data 2020 is recorded for each dish. The dish data 2020 isdesignated on the basis of a source image (Source), and an area (Area)that is cut away from the source image. That is, the dish data 2020 isdata of an image that is cut away from the source image, and registeredin association with information regarding the source image. For example,the pieces of dish data 2020 a and 2020 b are images obtained by cuttingaway different areas from the same source image (S001.jpg).Additionally, the source image is not limited to an image in the jpgformat. An image size of the dish data 2020 is normalized as 20×20=400(pixels), but another normalization method may also be used.Alternatively, the image size does not necessarily have to benormalized.

In the present embodiment, the data of the source image is stored in thedatabase 140. The dish data 2020 is referred to by cutting away adesignated area from the source image. As another example, an image ofthe dish data 2020 may be stored in the database 140 separately from thesource image.

FIG. 8 is a flowchart illustrating an example of a dish recognitionprocess in the second embodiment of the present disclosure. In theillustrated example, the dish recognition unit 130 first performs loopprocessing in units of a source image of the dish data 2020 whenrecognizing a dish included in a dish image (step S201).

The dish recognition unit 130 determines, with respect to each sourceimage, whether a dish image acquired by the image acquisition unit 110is similar to the source image (step S203). It may be determined whetherthe dish image is similar to the source image, for example, on the basisof a characteristic such as average values of luminance and averagevalues of colors of the respective images.

If the dish image is similar to the source image in step S203 (YES), thedish recognition unit 130 performs loop processing on the dish data 2020corresponding to the source image, which is the dish data 2020 that iscut away from the source image (step S205). In loop processing, the dishrecognition unit 130 uses the dish data 2020 for matching with the dishimage (step S207). To the contrary, if the dish image is not similar tothe source image (NO), the dish recognition unit 130 does not use thedish data 2020 corresponding to the source image for matching, andproceeds to process the next source image (step S201).

If strict similarity determination was performed on images in step S203,dish data corresponding to a source image that should not be actuallyexcluded would be excluded from matching targets because even if a dishimage and a source image are shot under the same shooting condition, theimages both may have different dishes. Thus, for example, a lowthreshold value for similarity may be set in the similaritydetermination on images in step S203 such that only dish data that iscut away from a source image shot under a shooting condition clearlydifferent from a shooting condition of the dish image is excluded frommatching targets.

According to the configuration as described in the present embodiment,it is possible to maintain the accuracy of dish recognition withoutacquiring metadata of a dish image or associating various types ofinformation with dish data, and decrease processing loads at the sametime.

3. Third Embodiment

FIG. 9 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to athird embodiment of the present disclosure. With reference to FIG. 9, aninformation processing apparatus 300 includes an image acquisition unit110, a dish recognition unit 130, a database 140, a result output unit150, and a dish data management unit 360. The information processingapparatus 300 may further include an information extraction unit 120.

As shown in the figure, the information processing apparatus 300 mayhave the same configuration described in the first or second embodiment,except the information processing apparatus 300 has the dish datamanagement unit 360 installed therein. Accordingly, a function of thedish data management unit 360 will be mainly described below, and theother repeated detailed description will be omitted.

The dish data management unit 360 registers dish data in an additionalmanner, and deletes a part of the dish data on the basis of apredetermined criterion if a predetermined volume of dish data areaccumulated. The dish data management unit 360 is realized by aprocessor such as a CPU operating in accordance with a program stored ina memory.

As described above, for example, dish data may be registered inaccordance with an input operation of a user, and may be distributed bya user who satisfies a predetermined condition. Then, the dish datamanagement unit 360 registers new dish data with the database 140 in anadditional manner. However, while dish data is registered in anadditional manner, the volume of dish data used by the dish recognitionunit 130 also unilaterally increases. Accordingly, processing loads indish recognition also increases, though the dish data is selectivelyreferred to under a predetermined condition. The dish data managementunit 360 therefore registers dish data in an additional manner, andautomatically deletes dish data on the basis of a predeterminedcriterion in the present embodiment.

FIG. 10 is a diagram illustrating an example of information that isassociated with dish data in the third embodiment of the presentdisclosure. FIG. 10 illustrates pieces of dish data 3020 a, 3020 b, 3020p, and 3020 q as examples of dish data 3020.

The dish data 3020 is registered for each dish. Additionally, items thatare used for the dish recognition unit 130 to select which dish data isreferred to are not shown in the figure because the items are the samedescribed in the first or second embodiment. The dish data 3020 isassociated with a date when the dish data 3020 last matches (Lastmatched), the number of matching (Match count), a date when the dishdata 3020 is last used (Last used), and a usage count (Usage count) inaddition to the above-described items. For example, the dish data 3020 bis associated with information indicating that the dish data 3020 b lastmatches with a dish image on Aug. 20, 2012, the dish data 3020 b hasmatches with a dish image 18 times, the dish data 3020 b is last usedfor matching on Oct. 11, 2012, and the dish data 3020 b has been usedfor matching 130 times.

In the present embodiment, the dish data management unit 360 uses suchkinds of information as indices to automatically delete the dish data3020, and maintains the volume of dish data 3020 within an appropriaterange. For example, the dish data management unit 360 first deletes datathat has an older Last matched, which namely means that a longer timehas passed since the data last matches with a dish image. If data isautomatically deleted on basis of this criterion, the dish data 3020 bis the first to be deleted and the dish data 3020 p is subsequentlydeleted in the illustrated example.

For example, the dish data management unit 360 may also first deletedata that has a smaller Match count, which namely means the data matcheswith a dish image fewer times. If data is automatically deleted on thebasis of this criterion, the dish data 3020 q is the first to be deletedand the dish data 3020 b is subsequently deleted in the illustratedexample. Similarly, the dish data management unit 360 may also firstdelete data that has an older Last used, or data that has a smallerUsage count.

In the illustrated example, if the dish data management unit 360 uses aLast matched or a Match count as a criterion for automatic deletion,data such as the dish data 3020 b that is frequently used for matchingbut less likely to match with a dish image is deleted. Meanwhile, if thedish data management unit 360 uses a Last used or a Match count as acriterion for automatic deletion, data such as the dish data 3020 q thatis rarely used but more likely to match with a dish image when used isdeleted.

Alternatively, the dish data management unit 360 may combine theabove-described indices as a criterion for automatic deletion. Forexample, the dish data management unit 360 may calculate an evaluationvalue for automatic deletion of the dish data 3020 by weighting andadding the multiple indices. In that case, the dish data 3020 that has,for example, a lower evaluation value is the first to be deleted.

The dish data management unit 360 may also use the multiple indices likea decision tree for automatic deletion. In that case, the dish datamanagement unit 360 first decides a deletion target on the basis of afirst index, and further decides a deletion target on the basis of adifferent second index among the pieces of dish data 3020 that comeunder the same first index. For example, in the illustrated example, iftwo of four pieces of dish data 3020 are deleted, and if, for example, aLast used is set as a first index, the dish data 3020 q is decided as adeletion target. However, the remaining pieces of dish data 3020 a, 3020b, and 3020 p come under the same index. Next, if, for example, a Usagecount is used as a second index, the dish data 3020 p can be decided asa deletion target among the three pieces of dish data.

The indices used for automatic deletion of dish data are not limited tothe above-described example. For example, dish data that periodicallymatches may be excluded from automatic deletion targets, as useful dataregardless of an index as described above. Data for recognizing a dishsuch as a seasonal dish that is not frequently served but is served inmost cases in a predetermined period can hereby be prevented from beingdeleted. Automatic deletion of the dish data 3020 performed by the dishdata management unit 360 may be performed through an input operation ofa user that manages the dish data 3020. For example, a user may protectspecific dish data 3020 through an input operation, and be able toexclude the specific dish data 3020 from automatic deletion targets. Tothe contrary, a user may compulsorily delete the specific dish data 3020through an input operation.

According to the configuration as described above in the presentembodiment, it is possible to improve the accuracy of dish recognitionby taking in new dish data, and prevent processing loads from increasingdue to infinite increase of dish data.

4. Fourth Embodiment

FIG. 11 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afourth embodiment of the present disclosure. With reference to FIG. 11,an information processing apparatus 400 includes an image acquisitionunit 110, a dish recognition unit 130, a database 140, a result outputunit 150, a standard dish recognition unit 470, and a recognition resultintegrating unit 480. The information processing apparatus 400 mayfurther include an information extraction unit 120 or a dish datamanagement unit 360.

As shown in the figure, the information processing apparatus 400 mayhave the same configuration described in the first to third embodiments,except the information processing apparatus 400 has the standard dishrecognition unit 470 and the recognition result integrating unit 480installed therein. Accordingly, functions of the standard dishrecognition unit 470 and the recognition result integrating unit 480will be mainly described below, and the other repeated detaileddescription will be omitted.

The standard dish recognition unit 470 recognizes a single or multipledishes included in a dish image acquired by the image acquisition unit110 without using a condition used for selecting dish data that isreferred to when the dish recognition unit 130 performs a recognitionprocess. A technique for a dish recognition process used by the standarddish recognition unit 470 is not particularly limited. For example, thestandard dish recognition unit 470 may identify a dish area in a dishimage through edge detection, and recognize a dish included in the areathrough learning-based object recognition. The standard dish recognitionunit 470 is realized by a processor such as a CPU operating inaccordance with a program stored in a memory.

A dish recognition process performed by the dish recognition unit 130uses dish data associated with, for example, information indicating asituation in which a user shoots a dish image, and informationindicating a situation in which a meal is taken, which namely means apersonalized process for each user. Meanwhile, the standard dishrecognition unit 470 uses common data for each user, and provides a morestandard and generalized dish recognition process. Independent of aperson relating to a dish image, a shooting environment of the dishimage, a shooting place of the dish image, and a shooting time of thedish image, the standard dish recognition unit 470 performs dishrecognition.

Thus, even if it is difficult for the dish recognition unit 130 toperform dish recognition, which namely means, for example, that nometadata is added to a dish image so that information is not acquired orthat the database 140 does not store dish data which satisfies acondition indicated by information such as metadata (for example, when auser takes a meal in a restaurant that the user has not visited before),the standard dish recognition unit 470 may successfully perform dishrecognition.

To the contrary, it is difficult for the standard dish recognition unit470 to perform proper dish recognition on a dish image shot in aparticular situation, which namely means, for example, that the dishimage is shot under a very dimly lit lamp in an atmospheric restaurantor that a dish is served at home in a uniquely shaped bowl. It is alsodifficult to recognize, for example, a user original dish, whose data isnot prepared in advance. Accordingly, for example, if necessaryinformation is acquired from metadata of a dish image, and if thedatabase 140 stores dish data that satisfies a condition indicated bythe information (for example, a user takes the same dishes twice or morein the same place), the dish recognition unit 130 effectively performsdish recognition.

Both of the dish recognition unit 130 and the standard dish recognitionunit 470 are therefore provided in the present embodiment, which allowsthe dish recognition unit 130 to accurately recognize a dish as taken bya user on a daily basis, and allows the standard dish recognition unit470 to recognize even a dish that the user takes for the first time.

The recognition result integrating unit 480 integrates a recognitionresult obtained by the dish recognition unit 130 with a recognitionresult obtained by the standard dish recognition unit 470. If one of thedish recognition unit 130 and the standard dish recognition unit 470succeeds in dish recognition, the recognition result integrating unit480 provides the recognition result to the result output unit 150. Ifboth of the dish recognition unit 130 and the standard dish recognitionunit 470 succeed in dish recognition, the recognition result integratingunit 480 adopts the recognition result, for example, in accordance withpriority that is set in advance, and provides the recognition result tothe result output unit 150. Alternatively, for example, if eachrecognition unit calculates a score indicating the accuracy of therecognition result, the recognition result integrating unit 480 mayadopt one of the recognition results that has a higher score. Therecognition result integrating unit 480 is also realized by a processorsuch as a CPU operating in accordance with a program stored in a memory.

Furthermore, if both of the dish recognition unit 130 and the standarddish recognition unit 470 succeed in dish recognition, and if both ofthe recognition results are the same, the recognition result integratingunit 480 may negatively evaluate dish data that is recognized by thedish recognition unit 130 as matching with a dish image. This is becausea dish recognized by using the dish data can also be recognized throughdish recognition generalized by the standard dish recognition unit 470so that the dish recognition unit 130 does not have to recognize thedish data. The negatively evaluated dish data becomes more likely to bedeleted, for example, through automatic deletion performed by the dishdata management unit 360.

5. Fifth Embodiment

FIG. 12 is a block diagram illustrating a schematic functionalconfiguration of an information processing apparatus according to afifth embodiment of the present disclosure. With reference to FIG. 12,an information processing apparatus 500 includes an image acquisitionunit 110, a dish recognition unit 130, a database 140, a result outputunit 150, a standard dish recognition unit 470, a recognition resultintegrating unit 480, and a dish data management unit 560. Theinformation processing apparatus 500 may further include an informationextraction unit 120.

As shown in the figure, the information processing apparatus 500 mayhave the same configuration described in the fourth embodiment, exceptthe information processing apparatus 500 has the dish data managementunit 560 installed therein. Accordingly, a function of the dish datamanagement unit 560 will be mainly described below, and the otherrepeated detailed description will be omitted.

The dish data management unit 560 registers dish data with the database140 in an additional manner on the basis of a recognition resultobtained by the standard dish recognition unit 470. In the presentembodiment, if a dish included in a dish image is not automaticallyrecognized, the recognition result is decided by the standard dishrecognition unit 470 on the basis of an input of a user. In that case,the dish data management unit 560 registers data of the dish, which isnot automatically recognized, as dish data in an additional manner. Aresult obtained by automatically recognizing a dish included in a dishimage may be corrected by the standard dish recognition unit 470 on thebasis of an input of a user. In that case, the dish data management unit560 registers the data of the dish having the recognition result, whichis decided through correction, as dish data in an additional manner. Thedish data management unit 560 is realized by a processor such as a CPUoperating in accordance with a program stored in a memory.

FIG. 13 is a diagram for describing an example of additionalregistration of dish data in the fifth embodiment of the presentdisclosure. FIG. 13 illustrates an example of an automatic recognitionresult obtained by the standard dish recognition unit 470. The resultindicates that the rice 1012 a, the egg 1012 c, the fish 1012 d, and themiso soup 1012 e are recognized as the dishes 1012 included in the dishimage 1010, but the salad 1012 b is not recognized. Although the misosoup 1012 e is certainly recognized, it is determined that the miso souphas a smaller dish area than the actual area for the miso soup.

Accordingly, a user corrects the recognized area, and manually inputsthe recognition result. For example, such an input is acquired when arecognition result obtained by the standard dish recognition unit 470 isprovided to the user via the recognition result integrating unit 480 andthe result output unit 150, and the user makes correction and performs amanual input operation. The figure illustrates the example in which auser uses a touch panel and the like to designate a position on animage, and designates an area for the displayed dish image 1010.However, the example of the user input is not limited thereto. Variousinput devices may be used for performing an input operation.

In the illustrated example, first, a user corrects the recognized areafor the miso soup 1012 e. The area for the miso soup 1012 e indicated bythe recognition result hereby matches with the area in which the actualmiso soup bowl is shown. Then, the dish data management unit 560registers the dish data corresponding to the corrected miso soup 1012 ewith the database 140 in an additional manner. This is because it ismore appropriate to use the dish data registered this time by the dishrecognition unit 130 in order to recognize the miso soup 1012 e includedin the dish image next time or later since the standard dish recognitionunit 470 fails to correctly recognize the area for the miso soup 1012 e.

Next, the user performs a manual input operation to cause the salad 1012b, which fails to be recognized, to be recognized. For example, the userdesignates the area for the salad 1012 b in the dish image 1010, andinputs a dish name, “salad,” for the designated area. The dish datamanagement unit 560 registers the dish data corresponding to the salad1012 b with the database 140 in an additional manner. This is because itis more appropriate to use the dish data registered this time by thedish recognition unit 130 in order to recognize the salad 1012 bincluded in the dish image next time or later since the standard dishrecognition unit 470 fails to recognize the salad 1012 b.

Additionally, the dish data management unit 560, which registers thedish data with the database 140 in an additional manner, mayautomatically delete dish data on basis of a predetermined criterion inthe same way as the dish data management unit 360 described in the thirdembodiment in order to prevent indefinite increase of dish data.

According to the configuration as described above in the presentembodiment, if a user does not explicitly perform a registration processon dish data, the dish data management unit 560 automatically registers,with the database 140, dish data regarding a dish that the standard dishrecognition unit 470 fails to correctly and automatically recognize.Thus, dishes that are recognized by the dish recognition unit 130 by useof the dish data stored in the database 140 can be limited to dishesthat are difficult for the standard dish recognition unit 470 torecognize. Consequently, only a minimum volume of dish data are storedin the database 140, which decreases processing loads in dishrecognition performed by the dish recognition unit 130.

6. Sixth Embodiment

FIG. 14 is a block diagram illustrating a schematic functionalconfiguration of a system according to a sixth embodiment of the presentdisclosure. With reference to FIG. 14, a system 60 includes a client 600and a server 700. The client 600 includes an image acquisition unit 110,a dish recognition unit 130, a database 140, a result output unit 150,and a recognition result integrating unit 480. The client 600 mayfurther include an information extraction unit 120 or a dish datamanagement unit 560. The server 700 includes a standard dish recognitionunit 770.

The present embodiment is different from the fourth or fifth embodimentin that the same functional configuration described in the fourth orfifth embodiment is distributed and realized in the client 600 and theserver 700. For example, the client 600 is realized by a terminalapparatus used by a user. The server 700 is realized by a single ormultiple server apparatuses that communicate with the terminalapparatus, which is the client 600, to provide a service to the user.Since the other parts of the configuration according to the presentembodiment are substantially the same described in the fourth or fifthembodiment, the repeated detailed description will be omitted.

The standard dish recognition unit 770 of the server 700 receives a dishimage from the image acquisition unit 110 of the client 600, andperforms the same standardized process of dish recognition performed bythe standard dish recognition unit 470 described in the fourth or fifthembodiment. The standard dish recognition unit 770 transmits therecognition result to the recognition result integrating unit 480 of theclient 600. That is, in the present embodiment, the client 600 asks theserver 700 to perform the standardized process of dish recognitionperformed by the standard dish recognition unit 770.

The information processing apparatuses 400 and 500 according to thefourth and fifth embodiments may also be terminal apparatuses (clients)and servers. That is, in both of the embodiments, a dish recognitionprocess performed by the dish recognition unit 130 and a dishrecognition process performed by the standard dish recognition unit 470are performed together in clients or servers. However, as describedabove, a process of dish recognition places heavy processing loads.Accordingly, it may not be practical that the dish recognition processesare performed together in clients that have relatively low processingcapacity. Thus, it is preferable in view of processing load adjustmentthat the dish recognition processes are performed together in serversthat have high processing capacity.

However, for example, as illustrated in the example of FIG. 3B, dishdata may include personal information of a user. Thus, it may not bepreferable to store dish data in a server. Accordingly, in the presentembodiment, privacy of a user is protected by disposing the database140, which stores dish data, and the dish recognition unit 130, whichperforms a dish recognition process by using the dish data, in theclient 600, while processing loads are decreased on the client 600 bycausing the standard dish recognition unit 770 disposed in the server700 to perform a standardized dish recognition process, which usesgeneral data.

7. Hardware Configuration

Next, with reference to FIG. 15, a hardware configuration of aninformation processing apparatus according to an embodiment of thepresent disclosure will be described. FIG. 15 is a block diagram fordescribing a hardware configuration of an information processingapparatus. An illustrated information processing apparatus 900 mayrealize, for example, the information processing apparatuses 100 to 500,the client 600, and the server 700 in the above-described embodiments.

The information processing apparatus 900 includes a central processingunit (CPU) 901, read only memory (ROM) 903, and random access memory(RAM) 905. The information processing apparatus 900 may include a hostbus 907, a bridge 909, an external bus 911, an interface 913, an inputdevice 915, an output device 917, a storage device 919, a drive 921, aconnection port 923, and a communication device 925. The informationprocessing apparatus 900 may further include an imaging device 933 and asensor 935 as necessary. The information processing apparatus 900 mayinclude a processing circuit such as a digital signal processor (DSP)instead of or along with the CPU 901.

The CPU 901 functions as an arithmetic processing device and a controldevice, and controls all or a part of operations in the informationprocessing apparatus 900 in accordance with various programs recorded onthe ROM 903, the RAM 905, the storage device 919, or a removablerecording medium 927. The ROM 903 stores, for example, a program and anoperation parameter used by the CPU 901. The RAM 905 temporarily stores,for example, a program used when the CPU 901 operates, and a parametervarying as necessary white the CPU 901 is operating. The CPU 901, theROM 903, and the RAM 905 are connected to each other by the host bus 907including an internal bus such as a CPU bus. Furthermore, the host bus907 is connected to the external bus 911 such as a peripheral componentinterconnect/interface (PCI) bus via the bridge 909.

The input device 915 is a device operated by a user such as a mouse, akeyboard, a touch panel, a button, a switch, and a lever. The inputdevice 915 may be a remote control device that uses, for example,infrared radiation and another type of radiowaves. Alternatively, theinput device 915 may be an external connection apparatus 929 such as amobile phone that corresponds to an operation of the informationprocessing apparatus 900. The input device 915 includes an input controlcircuit that generates input signals on the basis of information whichis input by a user to output the generated input signals to the CPU 901.A user inputs various types of data and indicates a processing operationto the information processing apparatus 900 by operating the inputdevice 915.

The output device 917 includes a device that can visually or audiblyreport acquired information to a user. The output device 917 may be, forexample, a display device such as a liquid crystal display (LCD), aplasma display panel (PDP), and an organic electro-luminescence (EL)display, an audio output device such as a speaker and a headphone, and aprinter. The output device 917 outputs a result obtained through aprocess performed by the information processing apparatus 900, in theform of video such as text and an image, or sounds such as voice andaudio sounds.

The storage device 919 is a device for data storage that is an exampleof a storage unit of the information processing apparatus 900. Thestorage device 919 includes, for example, a magnetic storage device suchas a hard disk drive (HDD), a semiconductor storage device, an opticalstorage device, or a magneto-optical storage device. The storage device919 stores a program and various types of data executed by the CPU 901,various types of data acquired from an external apparatus, and the like.

The drive 921 is a reader/writer for the removable recording medium 927such as a magnetic disk, an optical disc, a magneto-optical disk, and asemiconductor memory, and built in or externally attached to theinformation processing apparatus 900. The drive 921 reads outinformation recorded on the mounted removable recording medium 927, andoutputs the information to the RAM 905. The drive 921 writes the recordinto the mounted removable recording medium 927.

The connection port 923 is a port for directly connecting a device tothe information processing apparatus 900. The connection port 923 maybe, for example, a universal serial bus (USB) port, an IEEE1394 port,and a small computer system interface (SCSI) port. The connection port923 may also be, for example, an RS-232C port, an optical audioterminal, and a high-definition multimedia interface (HDMI) port.Various types of data may be exchanged between the informationprocessing apparatus 900 and the external connection apparatus 929 byconnecting the external connection apparatus 929 to the connection port923.

The communication device 925 is a communication interface including, forexample, a communication device for connection to a communicationnetwork 931. The communication device 925 may be, for example, a wiredor wireless local area network (LAN), Bluetooth (registered trademark),or a communication card for a wireless USB (WUSB). The communicationdevice 925 may also be, for example, a router for optical communication,a router for asymmetric digital subscriber line (ADSL), or a modem forvarious types of communication. For example, the communication device925 transmits and receives signals in the Internet or transits signalsto and receives signals from another communication device by using apredetermined protocol such as TCP/IP. The communication network 931 towhich the communication device 925 connects is a network establishedthrough wired or wireless connection. The communication network 931 is,for example, the Internet, a home LAN, infrared communication, radiocommunication, or satellite communication.

The imaging device 933 is a device that shoots a real space by using animage sensor such as a charge coupled device (CCD) and a complementarymetal oxide semiconductor (CMOS), and various members such as a lens forcontrolling image formation of a subject image onto the image sensor,and generates the shot image. The imaging device 933 may shoot a stillimage or a moving image.

The sensor 935 is various sensors such as an acceleration sensor, a gyrosensor, a geomagnetic sensor, an optical sensor, and a sound sensor. Thesensor 935 acquires information regarding a state of the informationprocessing apparatus 900 such as a posture of a housing of theinformation processing apparatus 900, and information regarding anenvironment surrounding the information processing apparatus 900 such asluminous intensity and noise around the information processing apparatus900. The sensor 935 may include a global positioning system (GPS) sensorthat receives GPS signals to measure latitude, longitude, and altitudeof the apparatus.

As above, the example of the hardware configuration of the informationprocessing apparatus 900 has been described. A general-purpose membermay be used for each structural element, or hardware dedicated to afunction of each structural element may also be used. The configurationmay be changed as necessary in accordance with the state of the art atthe time of working of the present disclosure.

8. Supplement

The embodiments of the present disclosure may include, for example, theinformation processing apparatus, the system, the information processingmethod executed by the information processing apparatus or the system,the program for causing the information processing apparatus tofunction, and the recording medium having the program recorded thereon.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus including:

an image acquisition unit configured to acquire a dish image obtained byshooting a single or multiple dishes; and

a first dish recognition unit configured to recognize the single ormultiple dishes included in the dish image with reference to dish dataselected, from dish data registered in advance, based on a conditionregarding at least one of a person relating to the dish image, ashooting environment of the dish image, a shooting place of the dishimage, and a shooting time of the dish image.

(2) The information processing apparatus according to (1), furtherincluding:

a recognition result integrating unit configured to integrate a firstrecognition result obtained by the first dish recognition unit with asecond recognition result obtained by a second dish recognition unit,the second dish recognition unit recognizing the single or multipledishes included in the dish image without using the condition.

(3) The information processing apparatus according to (2), furtherincluding:

a dish data management unit configured to register the dish data in anadditional manner based on the second recognition result.

(4) The information processing apparatus according to (3),

wherein, when the single or multiple dishes are not automaticallyrecognized, the second recognition result is decided based on an inputof a user, and

wherein the dish data management unit registers the dish data in anadditional manner based on the second recognition result decided basedon the input of the user.

(5) The information processing apparatus according to (3) or (4),

wherein, the second recognition result is a result obtained byautomatically recognizing the single or multiple dishes or a resultobtained by correcting the result based on information that is input bya user, and

wherein the dish data management unit registers the dish data in anadditional manner based on the second recognition result decided throughcorrection based on the information that is input by the user.

(6) The information processing apparatus according to any one of (2) to(5),

wherein the information processing apparatus is a client used by a userwho shoots the dish image,

wherein the image acquisition unit transmits the acquired dish image toa server that includes the second dish recognition unit, and

wherein the recognition result integrating unit receives the secondrecognition result from the server.

(7) The information processing apparatus according to any one of (1) to(6), further including:

a dish data management unit configured to register the dish data in anadditional manner, and to delete a part of the dish data based on apredetermined criterion when a predetermined volume of the dish data areaccumulated.

(8) The information processing apparatus according to (7),

wherein the first dish recognition unit recognizes the single ormultiple dishes by matching the dish image with the selected dish data.

(9) The information processing apparatus according to (8),

wherein the dish data management unit deletes a part of the dish databased on a number indicating how many times the dish data is used formatching with the dish image, or a date and time indicating when thedish data is used for matching with the dish image.

(10) The information processing apparatus according (8) or (9),

wherein the dish data management unit deletes a part of the dish databased on a number indicating how many times the dish data successfullymatches with the dish image, or a date and time indicating when the dishdata successfully matches the dish image.

(11) The information processing apparatus according to any one of (1) to(10),

wherein the image acquisition unit acquires the dish image that entailsmetadata, and

wherein the information processing apparatus further includes:

-   -   an information extraction unit configured to extract the        information regarding at least one of the person relating to the        dish image, the shooting environment of the dish image, the        shooting place of the dish image, and the shooting time of the        dish image from the metadata.        (12) The information processing apparatus according to (11),

wherein the information extraction unit extracts information regardingthe person relating to the dish image, and

wherein the first dish recognition unit recognizes the single ormultiple dishes with reference to the dish data registered inassociation with the person.

(13) The information processing apparatus according to (11) or (12),

wherein the information extraction unit extracts information regardingthe shooting environment of the dish image, and

wherein the first dish recognition unit recognizes the single ormultiple dishes by using the dish data registered in association withinformation indicating an environment common to the shootingenvironment.

(14) The information processing apparatus according to any one of (11)to (13),

wherein the information extraction unit extracts information regardingthe shooting place of the dish image, and

wherein the first dish recognition unit recognizes the single ormultiple dishes by using the dish data registered in association withthe shooting place.

(15) The information processing apparatus according to any one of (11)to (14),

wherein the information extraction unit extracts information regardingthe shooting time of the dish image, and

wherein the first dish recognition unit recognizes the single ormultiple dishes by using the dish data registered in association with atime zone including the shooting time.

(16) The information processing apparatus according to any one of (1) to(10),

wherein the first dish recognition unit treats a characteristic of awhole of the dish image as information regarding at least one of theshooting environment of the dish image, the shooting place of the dishimage, and the shooting time of the dish image.

(17) The information processing apparatus according to (16),

wherein the dish data is data of an image that is cut away from a sourceimage, and is registered in association with information regarding thesource image, and

wherein, when the characteristic of the whole of the dish image issimilar to a characteristic of the source image, the first dishrecognition unit recognizes the single or multiple dishes by using thedata of the image that is cut away from the source image.

(18) An information processing apparatus including:

acquiring a dish image obtained by shooting a single or multiple dishes;and

recognizing the single or multiple dishes included in the dish imagewith reference to dish data selected, from dish data registered inadvance, based on a condition regarding at least one of a personrelating to the dish image, a shooting environment of the dish image, ashooting place of the dish image, and a shooting time of the dish image.

What is claimed is:
 1. An information processing apparatus comprising:an image acquisition unit configured to acquire a dish image obtained byshooting a single or multiple dishes; and a first dish recognition unitconfigured to recognize the single or multiple dishes included in thedish image with reference to dish data selected, from dish dataregistered in advance, based on a condition regarding at least one of aperson relating to the dish image, a shooting environment of the dishimage, a shooting place of the dish image, and a shooting time of thedish image.
 2. The information processing apparatus according to claim1, further comprising: a recognition result integrating unit configuredto integrate a first recognition result obtained by the first dishrecognition unit with a second recognition result obtained by a seconddish recognition unit, the second dish recognition unit recognizing thesingle or multiple dishes included in the dish image without using thecondition.
 3. The information processing apparatus according to claim 2,further comprising: a dish data management unit configured to registerthe dish data in an additional manner based on the second recognitionresult.
 4. The information processing apparatus according to claim 3,wherein, when the single or multiple dishes are not automaticallyrecognized, the second recognition result is decided based on an inputof a user, and wherein the dish data management unit registers the dishdata in an additional manner based on the second recognition resultdecided based on the input of the user.
 5. The information processingapparatus according to claim 3, wherein, the second recognition resultis a result obtained by automatically recognizing the single or multipledishes or a result obtained by correcting the result based oninformation that is input by a user, and wherein the dish datamanagement unit registers the dish data in an additional manner based onthe second recognition result decided through correction based on theinformation that is input by the user.
 6. The information processingapparatus according to claim 2, wherein the information processingapparatus is a client used by a user who shoots the dish image, whereinthe image acquisition unit transmits the acquired dish image to a serverthat includes the second dish recognition unit, and wherein therecognition result integrating unit receives the second recognitionresult from the server.
 7. The information processing apparatusaccording to claim 1, further comprising: a dish data management unitconfigured to register the dish data in an additional manner, and todelete a part of the dish data based on a predetermined criterion when apredetermined volume of the dish data are accumulated.
 8. Theinformation processing apparatus according to claim 7, wherein the firstdish recognition unit recognizes the single or multiple dishes bymatching the dish image with the selected dish data.
 9. The informationprocessing apparatus according to claim 8, wherein the dish datamanagement unit deletes a part of the dish data based on a numberindicating how many times the dish data is used for matching with thedish image, or a date and time indicating when the dish data is used formatching with the dish image.
 10. The information processing apparatusaccording claim 8, wherein the dish data management unit deletes a partof the dish data based on a number indicating how many times the dishdata successfully matches with the dish image, or a date and timeindicating when the dish data successfully matches the dish image. 11.The information processing apparatus according to claim 1, wherein theimage acquisition unit acquires the dish image that entails metadata,and wherein the information processing apparatus further comprises: aninformation extraction unit configured to extract the informationregarding at least one of the person relating to the dish image, theshooting environment of the dish image, the shooting place of the dishimage, and the shooting time of the dish image from the metadata. 12.The information processing apparatus according to claim 11, wherein theinformation extraction unit extracts information regarding the personrelating to the dish image, and wherein the first dish recognition unitrecognizes the single or multiple dishes with reference to the dish dataregistered in association with the person.
 13. The informationprocessing apparatus according to claim 11, wherein the informationextraction unit extracts information regarding the shooting environmentof the dish image, and wherein the first dish recognition unitrecognizes the single or multiple dishes by using the dish dataregistered in association with information indicating an environmentcommon to the shooting environment.
 14. The information processingapparatus according to claim 11, wherein the information extraction unitextracts information regarding the shooting place of the dish image, andwherein the first dish recognition unit recognizes the single ormultiple dishes by using the dish data registered in association withthe shooting place.
 15. The information processing apparatus accordingto claim 11, wherein the information extraction unit extractsinformation regarding the shooting time of the dish image, and whereinthe first dish recognition unit recognizes the single or multiple dishesby using the dish data registered in association with a time zoneincluding the shooting time.
 16. The information processing apparatusaccording to claim 1, wherein the first dish recognition unit treats acharacteristic of a whole of the dish image as information regarding atleast one of the shooting environment of the dish image, the shootingplace of the dish image, and the shooting time of the dish image. 17.The information processing apparatus according to claim 16, wherein thedish data is data of an image that is cut away from a source image, andis registered in association with information regarding the sourceimage, and wherein, when the characteristic of the whole of the dishimage is similar to a characteristic of the source image, the first dishrecognition unit recognizes the single or multiple dishes by using thedata of the image that is cut away from the source image.
 18. Aninformation processing apparatus comprising: acquiring a dish imageobtained by shooting a single or multiple dishes; and recognizing thesingle or multiple dishes included in the dish image with reference todish data selected, from dish data registered in advance, based on acondition regarding at least one of a person relating to the dish image,a shooting environment of the dish image, a shooting place of the dishimage, and a shooting time of the dish image.